Learning Generative ConvNet with Continuous Latent Factors by Alternating Back-Propagation

The supervised learning of the discriminative convolutional neural network (ConvNet or CNN) is powered by back-propagation on the parameters. In this paper, we show that the unsupervised learning of a popular top-down generative ConvNet model with latent continuous factors can be accomplished by a learning algorithm that consists of alternatively performing back-propagation on both the latent factors and the parameters. The model is a non-linear generalization of factor analysis, where the high-dimensional observed data vector, such as an image, is assumed to be the noisy version of a vector generated by a non-linear transformation of a low-dimensional vector of continuous latent factors. Furthermore, it is assumed that these latent factors follow known independent distributions, such as standard normal distributions, and the non-linear transformation is assumed to be parametrized by a top-down ConvNet, which is capable of approximating the highly non-linear mapping from the latent factors to the image. We explore a simple and natural learning algorithm for this model that alternates between the following two steps: (1) inferring the latent factors by Langevin dynamics or gradient descent, and (2) updating the parameters of the ConvNet by gradient descent. Step (1) is based on the gradient of the reconstruction error with respect to the latent factors, which is readily available by back-propagation. We call this step inferential back-propagation. Step (2) is based on the gradient of the reconstruction error with respect to the parameters, and is also obtained by back-propagation. We refer to this step as learning back-propagation. The code for inferential back-propagation in (1) is actually part of the code for learning back-propagation in (2), and thus the inferential back-propagation in (1) is actually a by-product of the learning backpropagation in (2). We show that such an alternating back-propagation algorithm can learn realistic generative models of natural images and sounds.